CLARIN-NL - tokenisation

Tokenizer and Sentence Splitter

root — Mon, 08 Apr 2019 08:39:13 +0000

Description

detects word- and sentence boundaries in raw text

CLARIN National Project

not specified

CLARIN centre

BBAW: Berlin-Brandenburg Academy of Sciences and Humanities

Research domain

Linguistics

Linguistic Subject

Morpho-syntax

Syntax

Tool task

tokenisation

sentence splitting

Country

Germany

CMDI File Link

License

not specified

Inventory Scope

global

Distribution Type

not specified

Tokenizer - OpenNLP Project

root — Mon, 08 Apr 2019 08:39:02 +0000

Description

Tokenizer from the OpenNLP Project

CLARIN National Project

not specified

CLARIN centre

SfS: Uni-Tuebingen

Research domain

Linguistics

Linguistic Subject

Orthography

Tool task

tokenisation

Country

Germany

Tool Type

web service

Research Phase

Enriching Data

Tool status

production

Input Language

unclear

Access Contact

/mailto:/

Project Contact

/mailto:/

Creator Contact

/mailto:/

Source code

http://portal.clarin.nl/not%20specified

Resource

CMDI File Link

License

not specified

Inventory Scope

global

Distribution Type

not specified

Tokenizer/Sentences - Alpino

root — Mon, 08 Apr 2019 08:39:02 +0000

Description

Tokenizer and sentence splitter from Alpino.

CLARIN National Project

not specified

CLARIN centre

SfS: Uni-Tuebingen

Research domain

Linguistics

Linguistic Subject

Morpho-syntax

Syntax

Tool task

tokenisation

sentence splitting

Country

Germany

Tool Type

web service

Research Phase

Enriching Data

Tool status

development

Input Language

Dutch

Access Contact

/mailto:/

Project Contact

/mailto:/

Creator Contact

/mailto:/

Source code

http://portal.clarin.nl/not%20specified

Resource

CMDI File Link

License

not specified

Inventory Scope

global

Distribution Type

not specified

Tokenizer/Sentences - OpenNLP Project

root — Mon, 08 Apr 2019 08:39:02 +0000

Description

Tokenizer/sentences from the OpenNLP project. The 'newlineBounds' parameter treats newlines as a hard break (a sentence boundary).

CLARIN National Project

not specified

CLARIN centre

SfS: Uni-Tuebingen

Research domain

Linguistics

Linguistic Subject

Morpho-syntax

Syntax

Tool task

tokenisation

sentence splitting

Country

Germany

Tool Type

web service

Research Phase

Enriching Data

Tool status

production

Input Language

unclear

Access Contact

/mailto:/

Project Contact

/mailto:/

Creator Contact

/mailto:/

Source code

http://portal.clarin.nl/not%20specified

Resource

CMDI File Link

License

not specified

Inventory Scope

global

Distribution Type

not specified

Stanford Tokenizer

root — Mon, 08 Apr 2019 08:38:59 +0000

Description

Stanford Tokenizer is a an efficient, fast, deterministic tokenizer.

CLARIN National Project

not specified

CLARIN centre

SfS: Uni-Tuebingen

Research domain

Linguistics

Linguistic Subject

Orthography

Tool task

tokenisation

Country

Germany

Tool Type

web service

Research Phase

Enriching Data

Tool status

production

Input Language

unclear

Access Contact

/mailto:/

Project Contact

/mailto:/

Creator Contact

/mailto:/

Source code

http://portal.clarin.nl/not%20specified

Resource

CMDI File Link

License

not specified

Inventory Scope

global

Distribution Type

not specified

Tokenizer

root — Mon, 08 Apr 2019 08:38:55 +0000

CLARIN National Project

not specified

CLARIN centre

IMS: University of Stuttgart

Research domain

Linguistics

Linguistic Subject

Morpho-syntax

Syntax

Tool task

tokenisation

sentence splitting

Country

Germany

Tool Type

web service

Research Phase

Enriching Data

Tool status

production

Input Language

Czech

Slovenian

Hungarian

Italian

German

English

Access Contact

/mailto:/

Project Contact

/mailto:/

Creator Contact

/mailto:/

Source code

http://portal.clarin.nl/not%20specified

Resource

CMDI File Link

License

not specified

Inventory Scope

global

Distribution Type

not specified

OpeNER tokenizer

root — Mon, 08 Apr 2019 08:38:55 +0000

CLARIN National Project

not specified

CLARIN centre

ILC-CNR

Research domain

Linguistics

Linguistic Subject

Morpho-syntax

Syntax

Tool task

tokenisation

Country

Italy

Tool Type

web service

Research Phase

Enriching Data

Tool status

development

Input Language

Italian

Access Contact

/mailto:/

Project Contact

/mailto:/

Creator Contact

/mailto:/

Source code

http://portal.clarin.nl/not%20specified

Resource

CMDI File Link

License

not specified

Inventory Scope

global

Distribution Type

not specified

ReLDI tag+lemma+NER

root — Mon, 08 Apr 2019 08:38:55 +0000

CLARIN National Project

not specified

CLARIN centre

JSI

Research domain

Linguistics

Linguistic Subject

Morpho-syntax

Syntax

Tool task

tokenisation

part of speech tagging

lemmatisation

named entity recognition

Country

Slovenia

Tool Type

web service

Research Phase

Enriching Data

Tool status

development

Input Language

Slovenian

Serbian

Croatian

Access Contact

/mailto:/

Project Contact

/mailto:/

Creator Contact

/mailto:/

Source code

http://portal.clarin.nl/not%20specified

Resource

CMDI File Link

License

not specified

Inventory Scope

global

Distribution Type

not specified

ReLDI tag+lemma+parse

root — Mon, 08 Apr 2019 08:38:55 +0000

CLARIN National Project

not specified

CLARIN centre

JSI

Research domain

Linguistics

Linguistic Subject

Morpho-syntax

Syntax

Tool task

tokenisation

part of speech tagging

lemmatisation

parsing

Country

Slovenia

Tool Type

web service

Research Phase

Enriching Data

Tool status

development

Input Language

Slovenian

Serbian

Croatian

Access Contact

/mailto:/

Project Contact

/mailto:/

Creator Contact

/mailto:/

Source code

http://portal.clarin.nl/not%20specified

Resource

CMDI File Link

License

not specified

Inventory Scope

global

Distribution Type

not specified

Ucto

root — Mon, 08 Apr 2019 08:38:55 +0000

https://raw.githubusercontent.com/LanguageMachines/ucto/master/logo.svg

Title

Ucto Tokeniser

Description

Ucto tokenizes text files: it separates words from punctuation, and splits sentences. This is one of the first tasks for almost any Natural Language Processing application. Ucto offers several other basic preprocessing steps such as changing case that you can all use to make your text suited for further processing such as indexing, part-of-speech tagging, or machine translation. The tokeniser engine is language independent. By supplying language-specific tokenisation rules in an external configuration file a tokeniser can be created for a specific language. Ucto comes with tokenization rules for English, Dutch, French, Italian, and Swedish; it is easily extendible to other languages. It recognizes dates, times, units, currencies, abbreviations. It recognizes paired quote spans, sentences, and paragraphs. It produces UTF8 encoding and NFC output normalization, optionally accepts other encodings as input. Optional conversion to all lowercase or uppercase. Ucto supports FoLiA XML.