目錄 一個簡單的實現(xiàn)使用BSF(寬度優(yōu)先搜索)進(jìn)行實現(xiàn)使用DFA(Deterministic Finite Automaton)進(jìn)行實現(xiàn) 一個簡單的實現(xiàn)

主要是通過循環(huán)和replace的方式進(jìn)行敏感詞的替換

class NaiveFilter(): ’’’Filter Messages from keywords very simple filter implementation >>> f = NaiveFilter() >>> f.parse('filepath') >>> f.filter('hello sexy baby') hello **** baby ’’’ def __init__(self):self.keywords = set([]) def parse(self, path):for keyword in open(path): self.keywords.add(keyword.strip().decode(’utf-8’).lower()) def filter(self, message, repl='*'):message = str(message).lower()for kw in self.keywords: message = message.replace(kw, repl)return message使用BSF(寬度優(yōu)先搜索)進(jìn)行實現(xiàn)

對于搜索查找進(jìn)行了優(yōu)化，對于英語單詞，直接進(jìn)行了按詞索引字典查找。對于其他語言模式，我們采用逐字符查找匹配的一種模式。

BFS：寬度優(yōu)先搜索方式

class BSFilter: ’’’Filter Messages from keywords Use Back Sorted Mapping to reduce replacement times >>> f = BSFilter() >>> f.add('sexy') >>> f.filter('hello sexy baby') hello **** baby ’’’ def __init__(self):self.keywords = []self.kwsets = set([])self.bsdict = defaultdict(set)self.pat_en = re.compile(r’^[0-9a-zA-Z]+$’) # english phrase or not def add(self, keyword):if not isinstance(keyword, str): keyword = keyword.decode(’utf-8’)keyword = keyword.lower()if keyword not in self.kwsets: self.keywords.append(keyword) self.kwsets.add(keyword) index = len(self.keywords) - 1 for word in keyword.split():if self.pat_en.search(word): self.bsdict[word].add(index)else: for char in word:self.bsdict[char].add(index) def parse(self, path):with open(path, 'r') as f: for keyword in f:self.add(keyword.strip()) def filter(self, message, repl='*'):if not isinstance(message, str): message = message.decode(’utf-8’)message = message.lower()for word in message.split(): if self.pat_en.search(word):for index in self.bsdict[word]: message = message.replace(self.keywords[index], repl) else:for char in word: for index in self.bsdict[char]:message = message.replace(self.keywords[index], repl)return message使用DFA(Deterministic Finite Automaton)進(jìn)行實現(xiàn)

DFA即Deterministic Finite Automaton，也就是確定有窮自動機(jī)。使用了嵌套的字典來實現(xiàn)。

class DFAFilter(): ’’’Filter Messages from keywords Use DFA to keep algorithm perform constantly >>> f = DFAFilter() >>> f.add('sexy') >>> f.filter('hello sexy baby') hello **** baby ’’’ def __init__(self):self.keyword_chains = {}self.delimit = ’x00’ def add(self, keyword):if not isinstance(keyword, str): keyword = keyword.decode(’utf-8’)keyword = keyword.lower()chars = keyword.strip()if not chars: returnlevel = self.keyword_chainsfor i in range(len(chars)): if chars[i] in level:level = level[chars[i]] else:if not isinstance(level, dict): breakfor j in range(i, len(chars)): level[chars[j]] = {} last_level, last_char = level, chars[j] level = level[chars[j]]last_level[last_char] = {self.delimit: 0}breakif i == len(chars) - 1: level[self.delimit] = 0 def parse(self, path):with open(path,encoding=’UTF-8’) as f: for keyword in f:self.add(keyword.strip()) def filter(self, message, repl='*'):if not isinstance(message, str): message = message.decode(’utf-8’)message = message.lower()ret = []start = 0while start < len(message): level = self.keyword_chains step_ins = 0 for char in message[start:]:if char in level: step_ins += 1 if self.delimit not in level[char]:level = level[char] else:ret.append(repl * step_ins)start += step_ins - 1breakelse: ret.append(message[start]) break else:ret.append(message[start]) start += 1return ’’.join(ret)

到此這篇關(guān)于Python 敏感詞過濾的實現(xiàn)示例的文章就介紹到這了,更多相關(guān)Python 敏感詞過濾內(nèi)容請搜索好吧啦網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章希望大家以后多多支持好吧啦網(wǎng)！