我正在尝试在文件中写入数组字节，但我不能。 （我可以在文件系统中创建这个文件，并且连接正常）。任何帮助表示感谢
I'm trying to write a array bytes in a file but i can not. (I can create this file in FileSystem, and the conection is working). Any help is grateful
主要错误是：
- 文件
/test/1.txt 只能复制到 0 个节点，而不是 minReplication (=1)。有1datanode(s)运行和1node(s)被排除在本次操作中
createBlockOutputStream 异常。
我也尝试写一个 String 和它不可能的。
public void establece_connexion() {
try {
//System.setProperty("HADOOP_USER_NAME", "hduser"); //por si acaso lo necesitamos.
//UserGroupInformation ugi = UserGroupInformation.createRemoteUser(usuario_remoto);
Configuration conf = new Configuration();
conf.set("hadoop.job.ugi", usuario_remoto);
conf.set("fs.defaultFS", namenode); // namenode ="hdfs://" + ip + ":" + puerto
hdfsFileSystem = FileSystem.get(conf);
//hdfsFileSystem = FileSystem.get(new URI(namenode), new Configuration());
System.out.println("Se declara HDFS como sistema de archivos subyacente");
} catch (IOException e) {
e.printStackTrace();
System.out.println("Error en la conexión");
}
}
public void write() {
int numero = 1;
while (numero <= get_numero_ficheros_totales()-1) {
try {
System.out.println("Voy a empezar a escribir DE NUEVO");
//Path workingDir = hdfsFileSystem.getWorkingDirectory();
//Path archivo = new Path(hdfsFileSystem.getWorkingDirectory() + "/" + hdfsPath + "/" + numero + ".txt"); //creamos el nuevo fichero//
Path archivo = new Path(namenode + "//test//" + numero + ".txt");
System.out.println("ESTOY AQUI YA ");
//Hay una comprobacion de si si existe el ficehro que yo he creado
if (hdfsFileSystem.exists(archivo)) {
System.out.println("El fichero " + numero + " ya existe, no lo voy a crear de nuevo");
} else { //una vez que comprobamos que no existe el archivo, o ya esta borrado el archivo lo creamos o lo cremos de nuevo.
System.out.println("no existe el archivo" + numero);
FSDataOutputStream fsdata = hdfsFileSystem.create(archivo);
try {
byte[] arrayBytes = array_relleno();
//assert false;
fsdata.write(arrayBytes,0, arrayBytes.length); //esto sería sin escritura en zip//
//fsdata.writeUTF("hello");
//fsdata.writeUTF("hello");
System.out.println("Voy a empezar a escribir DE NUEVO");
System.out.println(fsdata);
fsdata.close();
} catch (Exception e) {
System.out.println("No he podido escribir en el fichero");
}
}
} catch(IOException e){
e.printStackTrace();
}
numero++;
}
}
错误：
Se declara HDFS como sistema de archivos subyacente
Voy a empezar a escribir DE NUEVO
ESTOY AQUI YA
no existe el archivo1
Voy a empezar a escribir DE NUEVO
org.apache.hadoop.hdfs.client.HdfsDataOutputStream@6ccdb29f
2019-08-26 09:45:21,849 INFO [Thread-4] hdfs.DFSClient (DFSOutputStream.java:createBlockOutputStream(1368)) - Exception in createBlockOutputStream
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1533)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1309)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1262)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:448)
2019-08-26 09:45:21,854 INFO [Thread-4] hdfs.DFSClient (DFSOutputStream.java:nextBlockOutputStream(1265)) - Abandoning BP-328389572-127.0.1.1-1566466459521:blk_1073742162_1338
2019-08-26 09:45:21,916 INFO [Thread-4] hdfs.DFSClient (DFSOutputStream.java:nextBlockOutputStream(1269)) - Excluding datanode DatanodeInfoWithStorage[127.0.0.1:50010,DS-9a5d1ada-fad9-4916-82c3-3c7125a6907a,DISK]
2019-08-26 09:45:21,925 WARN [Thread-4] hdfs.DFSClient (DFSOutputStream.java:run(557)) - DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /test/1.txt could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1576)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2045)
at org.apache.hadoop.ipc.Client.call(Client.java:1475)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy10.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy11.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1455)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1251)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:448)
No he podido escribir en el fichero
得到答案，问题解决
我一直在尝试将集群连接到安装了 hadoop 的另一台 linux (ubuntu 18.10) 计算机。我访问了 mi windows 计算机（都在同一个网络中）。
在文件 core-site.xml 中，我已将 "localhost" 更改为 0.0.0.0 以便从其他机器 (windows) 进行访问。有问题。我将 ip 更改为 192.168.3.37（你在哪里约会的 ip）并且它工作正常。然后重启namenode和datanode。
old-dated---------------:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://0.0.0.0:8020</value>
<description>Nombre del filesystem por defecto.</description>
</property>
</configuration>
up-dated---------------:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://192.168.3.37:8020</value>
<description>Nombre del filesystem por defecto.</description>
</property>
</configuration>
一开始就放localhost，但那不能留，必须要从另一台机器上访问
我想我被允许创建文件但不能写入它们，因为我想我访问了名称节点但是当我访问数据节点时我给出了错误
主要错误是：
- 文件
/test/1.txt只能复制到 0 个节点，而不是 minReplication (=1)。有1datanode(s)运行和1node(s)被排除在本次操作中 createBlockOutputStream异常。
我也尝试写一个 String 和它不可能的。
public void establece_connexion() {
try {
//System.setProperty("HADOOP_USER_NAME", "hduser"); //por si acaso lo necesitamos.
//UserGroupInformation ugi = UserGroupInformation.createRemoteUser(usuario_remoto);
Configuration conf = new Configuration();
conf.set("hadoop.job.ugi", usuario_remoto);
conf.set("fs.defaultFS", namenode); // namenode ="hdfs://" + ip + ":" + puerto
hdfsFileSystem = FileSystem.get(conf);
//hdfsFileSystem = FileSystem.get(new URI(namenode), new Configuration());
System.out.println("Se declara HDFS como sistema de archivos subyacente");
} catch (IOException e) {
e.printStackTrace();
System.out.println("Error en la conexión");
}
}
public void write() {
int numero = 1;
while (numero <= get_numero_ficheros_totales()-1) {
try {
System.out.println("Voy a empezar a escribir DE NUEVO");
//Path workingDir = hdfsFileSystem.getWorkingDirectory();
//Path archivo = new Path(hdfsFileSystem.getWorkingDirectory() + "/" + hdfsPath + "/" + numero + ".txt"); //creamos el nuevo fichero//
Path archivo = new Path(namenode + "//test//" + numero + ".txt");
System.out.println("ESTOY AQUI YA ");
//Hay una comprobacion de si si existe el ficehro que yo he creado
if (hdfsFileSystem.exists(archivo)) {
System.out.println("El fichero " + numero + " ya existe, no lo voy a crear de nuevo");
} else { //una vez que comprobamos que no existe el archivo, o ya esta borrado el archivo lo creamos o lo cremos de nuevo.
System.out.println("no existe el archivo" + numero);
FSDataOutputStream fsdata = hdfsFileSystem.create(archivo);
try {
byte[] arrayBytes = array_relleno();
//assert false;
fsdata.write(arrayBytes,0, arrayBytes.length); //esto sería sin escritura en zip//
//fsdata.writeUTF("hello");
//fsdata.writeUTF("hello");
System.out.println("Voy a empezar a escribir DE NUEVO");
System.out.println(fsdata);
fsdata.close();
} catch (Exception e) {
System.out.println("No he podido escribir en el fichero");
}
}
} catch(IOException e){
e.printStackTrace();
}
numero++;
}
}
错误：
Se declara HDFS como sistema de archivos subyacente
Voy a empezar a escribir DE NUEVO
ESTOY AQUI YA
no existe el archivo1
Voy a empezar a escribir DE NUEVO
org.apache.hadoop.hdfs.client.HdfsDataOutputStream@6ccdb29f
2019-08-26 09:45:21,849 INFO [Thread-4] hdfs.DFSClient (DFSOutputStream.java:createBlockOutputStream(1368)) - Exception in createBlockOutputStream
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1533)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1309)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1262)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:448)
2019-08-26 09:45:21,854 INFO [Thread-4] hdfs.DFSClient (DFSOutputStream.java:nextBlockOutputStream(1265)) - Abandoning BP-328389572-127.0.1.1-1566466459521:blk_1073742162_1338
2019-08-26 09:45:21,916 INFO [Thread-4] hdfs.DFSClient (DFSOutputStream.java:nextBlockOutputStream(1269)) - Excluding datanode DatanodeInfoWithStorage[127.0.0.1:50010,DS-9a5d1ada-fad9-4916-82c3-3c7125a6907a,DISK]
2019-08-26 09:45:21,925 WARN [Thread-4] hdfs.DFSClient (DFSOutputStream.java:run(557)) - DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /test/1.txt could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1576)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2045)
at org.apache.hadoop.ipc.Client.call(Client.java:1475)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy10.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy11.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1455)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1251)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:448)
No he podido escribir en el fichero
得到答案，问题解决
我一直在尝试将集群连接到安装了 hadoop 的另一台 linux (ubuntu 18.10) 计算机。我访问了 mi windows 计算机（都在同一个网络中）。
在文件 core-site.xml 中，我已将 "localhost" 更改为 0.0.0.0 以便从其他机器 (windows) 进行访问。有问题。我将 ip 更改为 192.168.3.37（你在哪里约会的 ip）并且它工作正常。然后重启namenode和datanode。
old-dated---------------:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://0.0.0.0:8020</value>
<description>Nombre del filesystem por defecto.</description>
</property>
</configuration>
up-dated---------------:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://192.168.3.37:8020</value>
<description>Nombre del filesystem por defecto.</description>
</property>
</configuration>
一开始就放localhost，但那不能留，必须要从另一台机器上访问
我想我被允许创建文件但不能写入它们，因为我想我访问了名称节点但是当我访问数据节点时我给出了错误