如何在 Numba cuda 中对字符串数组执行内核函数？

Question

我有一个从文件中读取的字符串数组，我想将文件的每一行与特定字符串进行比较。文件太大（大约 200 MB 行）

我已按照本教程进行操作 https://nyu-cds.github.io/python-numba/05-cuda/，但它并未准确说明如何处理 strings/characters 数组。

import numpy as np
from numba import cuda



@cuda.jit
def my_kernel(io_array):

    tx = cuda.threadIdx.x

    ty = cuda.blockIdx.x

    bw = cuda.blockDim.x

    pos = tx + ty * bw
    if pos < io_array.size:  # Check array boundaries
        io_array[pos]   # i want here to compare each line of the string array to a specific line

def main():
    a = open("test.txt", 'r')  # open file in read mode

    print("the file contains:")
    data = country = np.array(a.read())


    # Set the number of threads in a block
    threadsperblock = 32

    # Calculate the number of thread blocks in the grid
    blockspergrid = (data.size + (threadsperblock - 1)) // threadsperblock

    # Now start the kernel
    my_kernel[blockspergrid, threadsperblock](data)


    # Print the result
    print(data)

if __name__ == '__main__':
        main()

我有两个问题。

首先：如何将我想将文件的每一行与其进行比较的句子（字符串）发送到内核函数。（在io_array不影响线程计算）

第二：它如何处理字符串数组？当我运行上述代码

时出现此错误

this error is usually caused by passing an argument of a type that is unsupported by the named function.
[1] During: typing of intrinsic-call at test2.py (18)

File "test2.py", line 18:
def my_kernel(io_array):
    <source elided>
    if pos < io_array.size:  # Check array boundaries
        io_array[pos]   # do the computation

P.S我是 Cuda 新手，刚刚开始学习它。

Answer 1

首先是这个：

data = country = np.array(a.read())

并不像您认为的那样。它不会产生一个可以像这样索引的 numpy 数组：

io_array[pos]

如果你不相信我，只需在普通的 python 代码中尝试一下：

print(data[0])

你会得到一个错误。如果您需要这方面的帮助，请在 python 或 numpy 标签上提问。

所以我们需要一种不同的方法来从磁盘加载字符串数据。为简单起见，我选择使用 numpy.fromfile()。此方法将要求文件中的所有行都具有相同的宽度。我喜欢这个概念。如果要处理不同长度的行，则需要描述更多信息。

如果我们以这种方式开始，我们可以将数据作为字节数组加载，并使用它：

$ cat test.txt
the quick brown fox.............
jumped over the lazy dog........
repeatedly......................
$ cat t43.py
import numpy as np
from numba import cuda

@cuda.jit
def my_kernel(str_array, check_str, length, lines, result):

    col,line = cuda.grid(2)
    pos = (line*(length+1))+col
    if col < length and line < lines:  # Check array boundaries
        if str_array[pos] != check_str[col]:
            result[line] = 0

def main():
    a = np.fromfile("test.txt", dtype=np.byte)
    print("the file contains:")
    print(a)
    print("array length is:")
    print(a.shape[0])
    print("the check string is:")
    b = a[33:65]
    print(b)
    i = 0
    while a[i] != 10:
        i=i+1
    line_length = i
    print("line length is:")
    print(line_length)
    print("number of lines is:")
    line_count = a.shape[0]/(line_length+1)
    print(line_count)
    res = np.ones(line_count)
    # Set the number of threads in a block
    threadsperblock = (32,32)

    # Calculate the number of thread blocks in the grid
    blocks_x = (line_length/32)+1
    blocks_y = (line_count/32)+1
    blockspergrid = (blocks_x,blocks_y)
    # Now start the kernel
    my_kernel[blockspergrid, threadsperblock](a, b, line_length, line_count, res)


    # Print the result
    print("matching lines (match = 1):")
    print(res)

if __name__ == '__main__':
        main()
$ python t43.py
the file contains:
[116 104 101  32 113 117 105  99 107  32  98 114 111 119 110  32 102 111
 120  46  46  46  46  46  46  46  46  46  46  46  46  46  10 106 117 109
 112 101 100  32 111 118 101 114  32 116 104 101  32 108  97 122 121  32
 100 111 103  46  46  46  46  46  46  46  46  10 114 101 112 101  97 116
 101 100 108 121  46  46  46  46  46  46  46  46  46  46  46  46  46  46
  46  46  46  46  46  46  46  46  10]
array length is:
99
the check string is:
[106 117 109 112 101 100  32 111 118 101 114  32 116 104 101  32 108  97
 122 121  32 100 111 103  46  46  46  46  46  46  46  46]
line length is:
32
number of lines is:
3
matching lines (match = 1):
[ 0.  1.  0.]
$

如何在 Numba cuda 中对字符串数组执行内核函数？

How to do kernel function on an array of strings in Numba cuda?

python

cuda

numba