Search DaniWeb - tokenizer

Retrieval Augmented Generation with Hugging Face Models in LangChain 1 Month Ago by usmanmalik57 …=bnb_config, device_map={"":0}) tokenizer = AutoTokenizer.from_pretrained(model_id) ``` The…= pipeline("text-generation", model=model, tokenizer=tokenizer, max_new_tokens=1000) hf = HuggingFacePipeline(pipeline=pipe)… Summarizing YouTube Video Transcriptions Using Distil Whisper and LLM 1 Month Ago by usmanmalik57 …quot;automatic-speech-recognition", model=whisper_model, tokenizer=processor.tokenizer, feature_extractor=processor.feature_extractor, max_new_tokens=128, torch_dtype=…= temperature, do_sample=True) decoded = tokenizer.batch_decode(generated_ids) return decoded[0].split("… Re: Tokenizer 13 Years Ago by vijayan121 …for dropped delimiters). Tokenize with a tokenizer using this as the tokenizer function. For example: [CODE]#include …<iostream> #include <boost/tokenizer.hpp> int main() { std::string str = &…quot;This is, a test"; typedef boost::tokenizer< boost::char_separator<char> > tokenizer_type … Tokenizer 13 Years Ago by gregarion … s = "This is, a test"; tokenizer<> tok(s); for(tokenizer<>::iterator beg=tok.begin(); beg!=tok… Re: Tokenizer 13 Years Ago by Fbody … may want to search the documentation for the "boost::tokenizer" type. You probably need to provide an argument to… Tokenizer 13 Years Ago by gayansameera My project want to write tokenizer which can able to tokenize input C code (already written C file input to the system) and identify what are the keywords, variables, comments, conditions etc. (all syntax provide by C). any one can help me to do this. Re: Tokenizer with <p id> tags 17 Years Ago by vijayan121 … are a few links: [URL]http://www.boost.org/libs/tokenizer/index.html[/URL] [URL]http://www-eleves-isia.cma.fr…/documentation/BoostDoc/boost_1_29_0/libs/tokenizer/examples.cpp[/URL] you could also use the boost string… Tokenizer with <p id> tags 17 Years Ago by katerinaaa Hi, I would like to ask if anyone know how can I create a tokenizer for a txt file in C++. I find it difficult because there are not only words but there are also numbers and <p id> tags. I have attached the file that is needed to be tokenized. Could anyone help me ? Thanks a lot Re: tokenizer 18 Years Ago by freesoft_2000 Hi everyone, See the below thread for better understanding of the string tokenizer class [url]http://www.daniweb.com/techtalkforums/thread26132.html[/url] Another thing is that if you want to split up a string try looking up the java String class api Richard West Re: Tokenizer 13 Years Ago by Narue What have you done so far? We're not going to help you by doing it for you, but we'll be glad to help with any issues when you prove that you've make some kind of honest attempt. Re: Tokenizer 13 Years Ago by gayansameera [QUOTE=Narue;1550061]What have you done so far? We're not going to help you by doing it for you, but we'll be glad to help with any issues when you prove that you've make some kind of honest attempt.[/QUOTE] okey thanks Narue, I'm staring to write my translator (I'm still in initial step). I already finish scanning file line by line. I write … Re: Tokenizer 13 Years Ago by Narue [QUOTE]I already finish scanning file line by line.[/QUOTE] This is your first problem. [QUOTE]I write function to split sting using space.[/QUOTE] This is your second problem. You need to take the stream as a whole and recognize tokens as they come character by character. Why? Because the following is perfectly legal C: [code] int(a)=… Re: Tokenizer 13 Years Ago by gayansameera [QUOTE=Narue;1550091]This is your first problem. This is your second problem. You need to take the stream as a whole and recognize tokens as they come character by character. Why? Because the following is perfectly legal C: [code] int(a)=12345;a*\ =10; [/code] If you only read line by line, you'll have to somehow recognize tokens split… Re: Tokenizer 13 Years Ago by Narue [QUOTE]after get one character I must search that character or character set in all grammar rules[/QUOTE] You don't have to exhaustively search against all rules, just the ones that are valid for the current token string. For example, if you read "b", then there's no reason to check the operator or literal rules because you clearly have … Re: Tokenizer 13 Years Ago by gayansameera [QUOTE=Narue;1550114]You don't have to exhaustively search against all rules, just the ones that are valid for the current token string. For example, if you read "b", then there's no reason to check the operator or literal rules because you clearly have either an identifier or a keyword.[/QUOTE] Thanks Due I will tryout this If there … Re: string tokenizer 17 Years Ago by kashko … it, then you can do this with String Tokenizer or Stream Tokenizer String tokenizer splits the string into chunks using a delimiter. You… functionto validate ip address from a given string. using string tokenizer. ex:- 192.160.1.5 please send me the code… Re: string tokenizer 17 Years Ago by masijade … it, then you can do this with String Tokenizer or Stream Tokenizer String tokenizer splits the string into chunks using a delimiter. You…, I'm sorry, but what part of "using string tokenizer" did you not understand. I am quite sure the… string tokenizer 17 Years Ago by PrincessG Anyone please kindly help me!! I'm working on a project, trying to make use of string tokenizer in ASP.NET, can it be done? I then want to make use of string tokenizer to read till the file extension. (.doc) so that I can assign a icon image to the different file extensions. Re: Lexer- Tokenizer problem 16 Years Ago by vijayan121 …lexer") [ push_back_a( tokens, "LEX" ) ] | str_p("tokenizer") [ push_back_a( tokens, "TOK" ) ] | (+~space_p) …> test_cases = list_of ( "test daniweb lexer xyz tokenizer lexer" ) ( "daniweblexer tokenizerlexer abcd lexerlexer"… string tokenizer 16 Years Ago by dkwantee hello! I want to know about string tokenizer. i have a file of the following format:- Studentname StudentID Studentgrades each is separated by a tab. and the size of the file is unknown. I have to read each line using string tokenizer..Well, i dont know much about this, can someone help me? Re: string tokenizer 16 Years Ago by Narue Um, did you not understand what I said? C++ doesn't define a string tokenizer. Now, are you expected to use a specific tokenizer class, or will a more generic solution work? I can tell you how to tokenize a string, but it would be a waste of my time to do so if you really want to use some non-standard class that I don't know about. Re: String Tokenizer? 16 Years Ago by Alex Edwards …several peices based on the token. */ static my_string *tokenizer(char *value, char token){ int peices = 1;…ms << endl; first_library::my_string *trying = ms.tokenizer("letsztokenisedzthis!", 'z'); cout << &… String Tokenizer and arrays 12 Years Ago by fattyJ … an array. My problem is that im using a string tokenizer to delimit my commas and that put the student information… that I keep getting an error code with my string tokenizer [code] import java.util.StringTokenizer; import java.io.*; import java… Segmentation fault with custom string tokenizer using pointer arithmetic 11 Years Ago by jesskavidja Hey there, I'm having issues with a custom string tokenizer I'm using for an assignment. I've looked around … nothing. Whenever I run the program and it runs the tokenizer, I get a seg fault. I'm fairly new to… Re: Help...How do i use the Tokenizer? 19 Years Ago by aj.wh.ca … work... My question is do i have to use the tokenizer to separate the numbers... if it is possible how can… there are couple of ways you can do this. String Tokenizer is simplest for parsing. Once you have read the file… Re: string tokenizer 17 Years Ago by kashko Why bother with tokenizer, which is a pain and buggy in early versions of the JDK (which you will encounter in legacy systems). You just need substring(), and/or lastIndexOf() Re: string tokenizer 16 Years Ago by Narue Standard C++ doesn't have a string tokenizer, so the class you're using isn't likely to be well known. Perhaps you can be more specific about what's not working and provide some code? Re: string tokenizer 16 Years Ago by dkwantee i have not written anything yet..i wanted to get some help on string tokenizer. i know i have to use this to read my file but i dont know how. do you have any simple examples on it? Re: Java's String Tokenizer 13 Years Ago by skylover87 I understand that tokenizer can be used to remove trailing or beginning of a … : "Software Development Example oneWhiteSpace" Is it possible with tokenizer or is there any workable way around? Re: Segmentation fault with custom string tokenizer using pointer arithmetic 11 Years Ago by asrockw7 …-to-pointer and see that this is true. Within the tokenizer, what you can do is instead of offsetting using the…