无法广播输入数组

Question

我有一个 csv 数据，数据的第一列是 'label' 并且第一列之后的列到最后的 784 列包含图像 (28*28) 格式的表示。

我使用以下函数创建了一个 numpy 数组元组。

下一步是我尝试将此数据集拆分为所需的 80% /20% 拆分以进行训练和验证。为此，我使用 loadData() 方法如下。当我运行拆分函数时，出现错误 could not broadcast input array from shape (5851,784) into shape (5851) 错误。

我的问题是我只想将使用 load(filename) 生成的可用元组拆分为两个数据集。有什么帮助吗？

filename=dir_path+'train1.csv'
def load(filename):
    # read file into a list of rows
    with open(filename, 'rU') as csvfile:
        lines = csv.reader(csvfile, delimiter=',')
        rows = list(lines)

    # create empty numpy arrays of the required size
    data = np.empty((len(rows), len(rows[0])-1), dtype=np.float64)
    expected = np.empty((len(rows),), dtype=np.int64)

    # fill array with data from the csv-rows
    for i, row in enumerate(rows):
        data[i,:] = row[1:]
        expected[i] = row[0]

    training_data = data, expected
    return training_data

print load(filename)

结果

 (array([[ 0.,  0.,  0., ...,  0.,  0.,  0.],
           [ 0.,  0.,  0., ...,  0.,  0.,  0.],
           [ 0.,  0.,  0., ...,  0.,  0.,  0.],
           ..., 
           [ 0.,  0.,  0., ...,  0.,  0.,  0.],
           [ 0.,  0.,  0., ...,  0.,  0.,  0.],
           [ 0.,  0.,  0., ...,  0.,  0.,  0.]]), array([1, 1, 1, ..., 1, 1, 1]))

运行这个函数要拆分：

def loadData():
    train_data= load(train_name)
    #test_data= load(test_name)

    training_data,validation_data =np.split(train_data, [int(.8 * len(train_data))])

    return train_data

print loadData()

结果： 无法将输入数组从形状 (5851,784) 广播到形状 (5851)

SOLUTION:

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
train_name=dir_path+'train8.csv'

test_name=dir_path+'test8.csv'

def load(filename):
    # read file into a list of rows
    with open(filename, 'rU') as csvfile:
        lines = csv.reader(csvfile, delimiter=',')
        rows = list(lines)

    # create empty numpy arrays of the required size
    data = np.empty((len(rows), len(rows[0])-1), dtype=np.float64)
    expected = np.empty((len(rows),), dtype=np.int64)

    # fill array with data from the csv-rows
    for i, row in enumerate(rows):
        data[i,:] = row[1:]
        expected[i] = row[0]

    result_data = data, expected
    return result_data

def loadData():
    train_data= load(train_name)[0]
    labels= load(train_name)[1]
    test_data= load(test_name)

    x_train, x_test, y_train, y_test = train_test_split(train_data, labels, test_size=0.33)

    training_data = (x_train, y_train)
    validation_data=(x_test, y_test)

    return (training_data, validation_data, test_data)

这个方案会匹配mnist数据集

Answer 1

据我了解，您正在将一个由一个矩阵和一个数组（形状不同）组成的元组传递给 np.split，这就是您收到广播错误的原因。如果你给 np.split 一个单一的矩阵，它工作正常：

train_data = np.zeros((5000, 784))
labels = np.zeros(5000)

train,test = np.split(train_data, [int(0.8 * len(train_data))])
print "Train: {0}, Test: {1}".format(train.shape, test.shape)

这给出了以下输出：

Train: (4000, 784), Test: (1000, 784)

而如果你传递一个矩阵和数组的元组：

train_data = np.zeros((5000, 784))
labels = np.zeros(5000)

train,test = np.split((train_data,labels), [int(0.8 *len(train_data))])

您收到广播错误：

ValueError: could not broadcast input array from shape (5000,784) into shape (5000)

如果你想拆分一个数据集，包括它的标签，我建议使用像 scikit learn 中的 train_test_split（通过 pip install sklearn 可用）这样的东西，它可以处理观察和标签使用相同的函数：

import numpy as np
from sklearn.model_selection import train_test_split

def loadData():

    train_data = np.zeros((5000, 784))
    labels = np.zeros(5000)
    x_train, x_test, y_train, y_test = train_test_split(train_data, labels, test_size=0.22)

    print "Training samples: {0}, training labels: {1}".format(x_train.shape, y_train.shape)
    print "Validation samples: {0}, validation labels: {1}".format(x_test.shape, y_test.shape)

if __name__ == "__main__":
    loadData()

输出：

Training samples: (3900, 784), training labels: (3900,)
Validation samples: (1100, 784), validation labels: (1100,)

无法广播输入数组

could not broadcast input array

python

numpy

mnist