如何将维基百科 xml 转储导入 mongodb？

Question

我使用了这个 github php 代码

https://github.com/kodekrash/wikipedia.org-xmldump-mongodb

通过

获取数据集

wget -c http://wikipedia.c3sl.ufpr.br/enwiki/20150901/enwiki-20150901-pages-articles.xml.bz2

相当大 12GB

修改了相应的配置

$dsname = 'mongodb://wiki:wiki@localhost:27017/wikipedia';
$file = '../data/enwiki-20150901-pages-articles.xml.bz2';
$logpath = './';

和运行来自命令行

php wikipedia.org-xmldump-mongodb.php

得到错误：

    PHP Warning:  simplexml_load_string(): Entity: line 37: parser error : expected '>' in /home/username/wiki-project/wikipedia.org-xmldump-mongodb/wikipedia.org-xmldump-mongodb.php on line 73
PHP Warning:  simplexml_load_string(): </namespaces> in /home/username/wiki-project/wikipedia.org-xmldump-mongodb/wikipedia.org-xmldump-mongodb.php on line 73
PHP Warning:  simplexml_load_string():            ^ in /home/username/wiki-project/wikipedia.org-xmldump-mongodb/wikipedia.org-xmldump-mongodb.php on line 73
PHP Warning:  simplexml_load_string(): Entity: line 38: parser error : Premature end of data in tag namespace line 34 in /home/username/wiki-project/wikipedia.org-xmldump-mongodb/wikipedia.org-xmldump-mongodb.php on line 73
PHP Warning:  simplexml_load_string():  in /home/username/wiki-project/wikipedia.org-xmldump-mongodb/wikipedia.org-xmldump-mongodb.php on line 73
PHP Warning:  simplexml_load_string(): ^ in /home/username/wiki-project/wikipedia.org-xmldump-mongodb/wikipedia.org-xmldump-mongodb.php on line 73
PHP Warning:  simplexml_load_string(): Entity: line 38: parser error : Premature end of data in tag namespaces line 1 in /home/username/wiki-project/wikipedia.org-xmldump-mongodb/wikipedia.org-xmldump-mongodb.php on line 73
PHP Warning:  simplexml_load_string():  in /home/username/wiki-project/wikipedia.org-xmldump-mongodb/wikipedia.org-xmldump-mongodb.php on line 73
PHP Warning:  simplexml_load_string(): ^ in /home/username/wiki-project/wikipedia.org-xmldump-mongodb/wikipedia.org-xmldump-mongodb.php on line 73
Aborting. Unable to parse namespaces.

我已经安装了

php, mbstring, simpleXML, mongodb extensions and mongodb 2.69

的输出

php -m

是

[PHP Modules]
bcmath
bz2
calendar
Core
ctype
date
dba
dom
ereg
exif
fileinfo
filter
ftp
gettext
hash
iconv
json
libxml
mbstring
mhash
mongo
openssl
pcntl
pcre
PDO
Phar
posix
Reflection
session
shmop
SimpleXML
soap
sockets
SPL
standard
sysvmsg
sysvsem
sysvshm
tokenizer
wddx
xml
xmlreader
xmlwriter
zip
zlib

[Zend Modules]

有人知道这个错误吗？谢谢

Answer 1

您必须转义 > 字符。在脚本文件中，第 72 行替换此行：

$chunk = str_replace( [ 'letter">', '</namespace>' ], [ 'letter" name="', '" />' ], $chunk );

与：

$chunk = str_replace( [ 'letter"\>', '</namespace\>' ], [ 'letter" name="', '" /\>' ], $chunk );

这对我有用！

如何将维基百科 xml 转储导入 mongodb？

How to import wikipedia xml dump into mongodb?

php

xml

wikipedia

mongodb