tokenization
noun[U]
(UKusuallytokenisation)uk/ˌtəʊ.kən.aɪˈzeɪ.ʃən/us/ˌtoʊ.kən.aɪˈzeɪ.ʃən/tokenizationnoun[U](COMPUTING)
computing,languagespecialized
theprocessofdividingaseriesofcharacters(=letters,numbers, or othermarksorsignsused in writing orprinting)intoseparatetokens(=units)that have ameaning:
Tokenizationshowsthetextas aseriesoftokensorusefulmeaningfulunitsthat arelargerthanindividualcharacters, butsmallerthanphrasesandsentences.
The lowest-levelformofannotationis tokenisation.
computingspecialized
theprocessofreplacingaprivatepieceofdatawith atoken(= a differentpieceofdatathatrepresentsthe first one), inordertopreventprivateinformationbeingseenby someone who is notallowedto do so:
Tokenizationprotectssensitivecardholderdatabyremovingit frommerchantsystemsentirely.
- Orthography-oriented tokenization andsegmentationaretackledin themainby usingorthographichintssuch asblankspacesthatmarkthelimitsof words.
- Forlanguagessuch asChineseandJapanese, the use of adictionaryforsentencetokenisation andsegmentationisessential.
- Theintegratedtokenizationtechnologymeansthat nocardholderdataisstored,transmittedorprocessedat theproperty.
Computer concepts
- 3-D printing
- adaptive learning
- additive manufacturing
- addy
- aeroplane mode
- backup
- capture
- computing
- configuration
- importation
- information age
- information security
- information superhighway
- infosec
- scan
- transcoding
- Trojan horse
- tuple
- Turing Machine
- Turing test
You can also find related words, phrases, and synonyms in the topics:
Codes & decoding
tokenizationnoun[U](PERSON)
theactof doing something thatseemstosupportorhelpagroupofpeoplewho aretreatedunfairlyinsociety, such as giving amemberof thatgroupanimportantorpublicposition, but that does not makechangesthat wouldhelpthatgroupofpeoplein alastingway:
Researchshowsthatgroupswhoexperienceminoritystatusoftenencountertokenisation.
Compare
tokenism
- There has been a tokenisation ofminorityrightswithout arealempowermentofminoritiesthemselves.
- Tokenization oftenresultsintokenemployeesreportingasfeelinglonelyanddisconnectedfrom themajorityemployees.
- By tokenization, animpressioniscreatedthat abusinessisinclusiveanddiverse.
Unfairness and favouring someone unfairly
- ableism
- ableist
- agism
- agist
- Anglocentric
- discriminate
- discrimination
- discriminatory
- elitism
- elitist
- one-sided
- one-sidedness
- one-way
- othering
- the old school tie
- unbalance
- unequal
- unequally
- uneven
- unevenly