Mosestokenizer 问题：[WinError 2] 系统找不到指定的文件

Question

无法弄清楚为什么会出现此问题。

from mosestokenizer import MosesDetokenizer

with MosesDetokenizer('en') as detokenize:
    print(detokenize(["hi", 'my', 'name', 'is', 'artem']))

这是我得到的：

stdbuf was not found; communication with perl may hang due to stdio buffering.
Traceback (most recent call last):
  File "C:\Users\ArtemLaptiev\Documents\GitHub\temp\foo.py", line 3, in <module>
    with MosesDetokenizer('en') as detokenize:
  File "C:\ProgramFiles\Anaconda\lib\site-packages\mosestokenizer\detokenizer.py", line 47, in __init__
    super().__init__(argv)
  File "C:\ProgramFiles\Anaconda\lib\site-packages\toolwrapper.py", line 52, in __init__
    self.start()
  File "C:\ProgramFiles\Anaconda\lib\site-packages\toolwrapper.py", line 92, in start
    cwd=self.cwd
  File "C:\ProgramFiles\Anaconda\lib\subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "C:\ProgramFiles\Anaconda\lib\subprocess.py", line 997, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified

感谢您的帮助！

Answer 1

使用 sacremoses 而不是 moses。

pip install -U sacremoses

和

from sacremoses import MosesTokenizer, MosesDetokenizer
with MosesDetokenizer() as detokenize:
    print(detokenize(["hi", 'my', 'name', 'is', 'artem']))

了解完整详情sacremoses

Mosestokenizer 问题：[WinError 2] 系统找不到指定的文件

Mosestokenizer issue: [WinError 2] The system cannot find the file specified

python

nlp

tokenize

nltk

anaconda