0
對於lucene索引(v6.1)快速創建,我想將Slick 3.1(Scala)中的數據拆分爲任意部分(塊),以便在線程中傳遞不同的數據集以加速索引過程。我在Scala中編寫了以下代碼來從MySQL獲取數據。如何將Slick 3.1(Scala)中的數據拆分爲4部分
class NotesService(val databaseService: DatabaseService)(implicit executionContext: ExecutionContext) extends NoteEntityTable {
import databaseService._
import databaseService.driver.api._
import com.github.t3hnar.bcrypt._
def getNotes(): Future[Seq[NoteEntity]] = db.run(notes.result)
}
case class NoteEntity(id: Option[Long] = None, title: String, teaser: String, description: String)
代碼NotesService
class NotesService(val databaseService: DatabaseService)(implicit executionContext: ExecutionContext) extends NoteEntityTable {
import databaseService._
import databaseService.driver.api._
import com.github.t3hnar.bcrypt._
def getNotes(): Future[Seq[NoteEntity]] = db.run(notes.result)
}
要提取NotesService數據我用:
def setI = {
val NUM_THREADS = Runtime.getRuntime().availableProcessors()
val IndexStoreDir = Paths.get("/var/www/html/Index")
val analyzer = new StandardAnalyzer()
val writerConfig = new IndexWriterConfig(analyzer)
writerConfig.setOpenMode(OpenMode.CREATE_OR_APPEND)
writerConfig.setRAMBufferSizeMB(500)
.setMaxBufferedDocs(2)
.setMergeScheduler(new ConcurrentMergeScheduler())
val directory = FSDirectory.open(IndexStoreDir)
var writer = new IndexWriter(directory, writerConfig)
val threads = Array.ofDim[IndexTh](NUM_THREADS)
val notes = notesService.getNotes()
for (i <- 0 until NUM_THREADS){
threads(i) = new IndexTh(notesService, writer)
//here on this line I want to pass different sets of data to thread.
}
for (i <- 0 until NUM_THREADS) {
threads(i).start()
println("Thread " + i + " Started!")
}
}
這裏就行:
threads(i) = new IndexTh(notesService, writer)
我怎麼可以拆分來自notesService的數據傳遞給threa d? 如何將筆記中的數據分成多個塊? 我想這樣的數據:
假設notesService.getNotes()
檢索20000行數據。現在我想將這些行分成4000行的5部分,這樣每4000行數據就可以傳遞給不同的線程。
你是什麼意思塊?分頁? – Roman
我想通過不同的數據集(拆分主要數據集)在多個線程 – Sujit
嗯我還是不完全明白問題是什麼。也許你需要輸出的例子會有幫助。 – Roman