tokenize
verb[T]
(UKusuallytokenise)uk/ˈtəʊ.kən.aɪz/us/ˈtoʊ.kən.aɪz/tokenizeverb[T](COMPUTING)
computing,languagespecialized
todivideaseriesofcharacters(=letters,numbers, or othermarksorsignsused in writing orprinting)intoseparatetokens(=units)that have ameaning:
Thestreamofcharactersin anaturallanguagetextmust be tokenized (brokenup intodistinctmeaningfulunits) before anylanguageprocessingcanhappen.
computingspecialized
toreplaceaprivatepieceofdatawith atoken(= a differentpieceofdatathatrepresentsthe first one), inordertopreventprivateinformationbeingseenby someone who is notallowedto do so:
When wereceivethetransactionsecurely, we tokenize it andsendback atokento themerchantso they never have toseeacardnumberforsettlement.
- First,yourscriptmust be tokenised andconvertedintomeaningfulunits.
- Use thehugearchiveofpreviouscustomersearchesand tokenize these into words.
- Tokenising acardtransactionisconsideredsaferas theactualcarddetailsare notshared.
Computer concepts
- 3-D printing
- adaptive learning
- additive manufacturing
- addy
- aeroplane mode
- backup
- capture
- computing
- configuration
- importation
- information age
- information security
- information superhighway
- infosec
- scan
- transcoding
- Trojan horse
- tuple
- Turing Machine
- Turing test
You can also find related words, phrases, and synonyms in the topics:
Codes & decoding
tokenizeverb[T](PERSON)
to do something thatseemstosupportorhelpagroupofpeoplewho aretreatedunfairlyinsociety, such as giving amemberof thatgroupanimportantorpublicposition, but that does not makechangesthat wouldhelpthatgroupofpeoplein alastingway:
He did notfeelas if he had been tokenized while atcollegebut had beenheldto the samestandardsas everyoneelse.
- Manyemployeesfeelthat they are being tokenised inpublicitymaterials, whiletheirconcernsaboutbiasandinequalityare notactuallybeingaddressed.
- How can youtellif you're being tokenized atyourworkplace?
- Theylisted8wayspeopleofcolourare tokenized in nonprofits.
Unfairness and favouring someone unfairly
- ableism
- ableist
- agism
- agist
- Anglocentric
- discriminate
- discrimination
- discriminatory
- elitism
- elitist
- one-sided
- one-sidedness
- one-way
- othering
- the old school tie
- unbalance
- unequal
- unequally
- uneven
- unevenly