東方算程譚

Oriental Code Talk ── επιστημηが与太をこく、弾幕とは無縁のシロモノ。

目次

Blog 利用状況

ニュース

著作とお薦めの品々は

著作とお薦めの品々は
東方熱帯林へ。

あわせて読みたい

わんくま

  1. 東京勉強会#2
    C++/CLI カクテル・レシピ
  2. 東京勉強会#3
    template vs. generics
  3. 大阪勉強会#6
    C++むかしばなし
  4. 東京勉強会#7
    C++むかしばなし
  5. 東京勉強会#8
    STL/CLRによるGeneric Programming
  6. TechEd 2007 @YOKOHAMA
    C++・C++/CLI・C# 適材適所
  7. 東京勉強会#14
    Making of BOF
  8. 東京勉強会#15
    状態遷移
  9. 名古屋勉強会#2
    WinUnit - お気楽お手軽UnitTest

CodeZine

  1. Cで実現する「ぷちオブジェクト指向」
  2. CUnitによるテスト駆動開発
  3. SQLiteで組み込みDB体験(2007年版)
  4. C++/CLIによるCライブラリの.NET化
  5. C# 1.1からC# 3.0まで~言語仕様の進化
  6. BoostでC++0xのライブラリ「TR1」を先取りしよう (1)
  7. BoostでC++0xのライブラリ「TR1」を先取りしよう (2)
  8. BoostでC++0xのライブラリ「TR1」を先取りしよう (3)
  9. BoostでC++0xのライブラリ「TR1」を先取りしよう (4)
  10. BoostでC++0xのライブラリ「TR1」を先取りしよう (5)
  11. C/C++に対応した、もうひとつのUnitTestFramework ─ WinUnit
  12. SQLiteで"おこづかいちょう"
  13. STL/CLRツアーガイド
  14. マージ・ソート : 巨大データのソート法
  15. ヒープソートのアルゴリズム
  16. C++0xの新機能「ラムダ式」を次期Visual Studioでいち早く試す
  17. .NETでマンデルブロ集合を描く
  18. .NETでマンデルブロ集合を描く(後日談)
  19. C++/CLI : とある文字列の相互変換(コンバージョン)
  20. インテルTBBによる選択ソートの高速化
  21. インテルTBB3.0 によるパイプライン処理
  22. Visual C++ 2010に追加されたSTLアルゴリズム
  23. Visual C++ 2010に追加されたSTLコンテナ「forward_list」
  24. shared_ptrによるObserverパターンの実装
  25. .NETでマンデルブロ集合を描く(番外編) ── OpenCLで超並列コンピューティング
  26. StateパターンでCSVを読む
  27. 状態遷移表からStateパターンを自動生成する
  28. 「ソートも、サーチも、あるんだよ」~標準C++ライブラリにみるアルゴリズムの面白さ
  29. インテルTBBの同期メカニズム
  30. なぜsetを使っちゃいけないの?
  31. WPFアプリケーションで腕試し ~C++でもWPFアプリを
  32. C++11 : スレッド・ライブラリひとめぐり
  33. Google製のC++ Unit Test Framework「Google Test」を使ってみる
  34. メールでデータベースを更新するココロミ
  35. Visitorパターンで遊んでみたよ
  36. Collection 2題:「WPFにバインドできる辞書」と「重複を許す検索set」
  37. Visual C++ 2012:stateless-lambdaとSQLiteのぷち拡張
  38. 「Visual C++ Compiler November 2012 CTP」で追加された6つの新機能

@IT

  1. Vista時代のVisual C++の流儀(前編)Vista到来。既存C/C++資産の.NET化を始めよう!
  2. Vista時代のVisual C++の流儀(中編)MFCから.NETへの実践的移行計画
  3. Vista時代のVisual C++の流儀(後編) STL/CLRによるDocument/Viewアーキテクチャ
  4. C++開発者のための単体テスト入門 第1回 C++開発者の皆さん。テスト、ちゃんとしていますか?
  5. C++開発者のための単体テスト入門 第2回 C++アプリケーションの効率的なテスト手法(CppUnit編)
  6. C++開発者のための単体テスト入門 第3回 C++アプリケーションの効率的なテスト手法(NUnit編)

AWARDS


Microsoft MVP
for Visual Developer - Visual C++


Wankuma MVP
for いぢわる C++


Nyantora MVP
for こくまろ中国茶

Xbox

Links

記事カテゴリ

書庫

日記カテゴリ

トークン分割

「std::stringにはstrtokみたいなのないの?」 って訊かれた。
ないのよねー ^^;

いいもんいいもん、作ればあるもん! ってんで作ってみた。
# boost::tokenizer はもっと高機能です

----- tokenizer.h -----
#ifndef TOKENIZER_H__
#define TOKENIZER_H__

#include <string>  // string
#include <utility> // pair

template<typename charT, typename traits=std::char_traits<charT>,
         typename alloc=std::allocator<charT> >
class tokenizer {
public:
  typedef std::basic_string<charT,traits,alloc> string_type;
  typedef std::pair<typename string_type::size_type,
                    typename string_type::size_type> range_type;
  // コンストラクタ: 区切り文字の集合を引数に与える
  explicit tokenizer(const string_type& delim) : delim_(delim) {}
  // 与えた文字列のトークン分割を開始する
  void start(const string_type& source) { source_ = source; reset(); }
  // startに与えた文字列のトークン分割を最初からやりなおす
  void reset() { range_.first = 0; range_.second = 0; }
  // トークンを切り出す。切り出すトークンがなければfalseを返す
  bool next() {
    range_.first =
      source_.find_first_not_of(delim_, range_.first + range_.second);
    if ( range_.first == std::string::npos ) {
      return false;
    }
    range_.second = source_.find_first_of(delim_, range_.first);
    if ( range_.second == std::string::npos ) {
      range_.second = source_.size() - range_.first;
      return true;
    }
    range_.second -= range_.first;
    return true;
  }
  // nextで切り出されたトークンを返す
  string_type token() const
    { return source_.substr(range_.first, range_.second); }
  // next切り出されたトークンの位置と長さを返す
  range_type range() const { return range_; }
private:
  string_type source_; // 分割対象となる文字列
  string_type delim_;  // 区切り文字の集合
  range_type range_;   // トークンの位置と長さ
};

#endif

---- つかいかた -----
#include <iostream>
#include "tokenizer.h"

int main() {
  tokenizer<char> tk(",.");
  tk.start("Hello,world.");
  while ( tk.next() ) {
    std::cout << tk.token() << std::endl;
  }
}

投稿日時 : 2007年9月11日 17:13

コメントを追加

# re: トークン分割 2007/09/11 17:43 fufuhehume

vectorに切り出してくれたほうが便利じゃないっすか?
ランダムアクセスできるから、来るデータの形式が固定ならコーディングがいろいろ楽になりそうですし。
というかSplit

# re: トークン分割 2007/09/11 17:50 επιστημη

あはーん、vectorっちゅーか

template<typename OutputIterator>
OutputIterator
 split(const string_type& source,OutputIterator to);
とかそんな感じになるんかしらね。

# re: トークン分割 2007/09/11 22:04 Imabeppu

to に、たとえば vector の iterator を指定するということですよね。呼び出し側が箱を用意しないといけないですかね。いくつ用意したらよいか……。

# re: トークン分割 2007/09/11 22:13 επιστημη

back_inserterでくるめばえぇです。
# (そのに)参照。

# re: トークン分割 2007/09/12 9:53 Imabeppu

なるほど。back_inserter 忘れてました。

# Greetings! Very helpful advice within this article! It is the little changes that produce the biggest changes. Thanks for sharing! 2021/08/31 21:06 Greetings! Very helpful advice within this article

Greetings! Very helpful advice within this article!
It is the little changes that produce the biggest changes.
Thanks for sharing!

# Greetings! Very helpful advice within this article! It is the little changes that produce the biggest changes. Thanks for sharing! 2021/08/31 21:07 Greetings! Very helpful advice within this article

Greetings! Very helpful advice within this article!
It is the little changes that produce the biggest changes.
Thanks for sharing!

# Greetings! Very helpful advice within this article! It is the little changes that produce the biggest changes. Thanks for sharing! 2021/08/31 21:08 Greetings! Very helpful advice within this article

Greetings! Very helpful advice within this article!
It is the little changes that produce the biggest changes.
Thanks for sharing!

# Greetings! Very helpful advice within this article! It is the little changes that produce the biggest changes. Thanks for sharing! 2021/08/31 21:09 Greetings! Very helpful advice within this article

Greetings! Very helpful advice within this article!
It is the little changes that produce the biggest changes.
Thanks for sharing!

# What's up mates, its great post regarding tutoringand fully defined, keep it up all the time. 2021/09/06 7:10 What's up mates, its great post regarding tutoring

What's up mates, its great post regarding tutoringand fully defined, keep it up all the time.

# What's up mates, its great post regarding tutoringand fully defined, keep it up all the time. 2021/09/06 7:11 What's up mates, its great post regarding tutoring

What's up mates, its great post regarding tutoringand fully defined, keep it up all the time.

# What's up mates, its great post regarding tutoringand fully defined, keep it up all the time. 2021/09/06 7:12 What's up mates, its great post regarding tutoring

What's up mates, its great post regarding tutoringand fully defined, keep it up all the time.

# What's up mates, its great post regarding tutoringand fully defined, keep it up all the time. 2021/09/06 7:13 What's up mates, its great post regarding tutoring

What's up mates, its great post regarding tutoringand fully defined, keep it up all the time.

# What's up, the whole thing is going sound here and ofcourse every one is sharing facts, that's in fact fine, keep up writing. 2021/10/26 1:01 What's up, the whole thing is going sound here and

What's up, the whole thing is going sound here and ofcourse every one is
sharing facts, that's in fact fine, keep up writing.

# Simply just had to tell you I am just thrilled I came on the page. 2021/11/03 16:28 Simply just had to tell you I am just thrilled I c

Simply just had to tell you I am just thrilled
I came on the page.

# Pretty! This has been a really wonderful article. Thanks for providing this info. 2021/11/12 11:41 Pretty! This has been a really wonderful article.

Pretty! This has been a really wonderful article.
Thanks for providing this info.

# Pretty! This has been a really wonderful article. Thanks for providing this info. 2021/11/12 11:42 Pretty! This has been a really wonderful article.

Pretty! This has been a really wonderful article.
Thanks for providing this info.

# Pretty! This has been a really wonderful article. Thanks for providing this info. 2021/11/12 11:43 Pretty! This has been a really wonderful article.

Pretty! This has been a really wonderful article.
Thanks for providing this info.

# Pretty! This has been a really wonderful article. Thanks for providing this info. 2021/11/12 11:44 Pretty! This has been a really wonderful article.

Pretty! This has been a really wonderful article.
Thanks for providing this info.

# It's in point of fact a great and useful piece of information. I'm satisfied that you shared this useful info with us. Please stay us up to date like this. Thanks for sharing. 2021/11/21 9:14 It's in point of fact a great and useful piece of

It's in point of fact a great and useful piece of information. I'm satisfied that you shared this useful info with us.
Please stay us up to date like this. Thanks for sharing.

# I think everything published made a lot of sense. However, what about this? suppose you were to create a awesome headline? I ain't saying your content is not good, but suppose you added a title that grabbed people's attention? I mean トークン分割 is kinda pla 2022/03/25 6:03 I think everything published made a lot of sense.

I think everything published made a lot of sense.

However, what about this? suppose you were to create a awesome headline?
I ain't saying your content is not good, but suppose you
added a title that grabbed people's attention? I mean トークン分割 is kinda plain. You might peek at Yahoo's front page and see
how they create news titles to grab people to click.

You might add a related video or a related picture or two
to grab people interested about everything've
written. In my opinion, it could bring your posts a little livelier.

# Good web site you've got here.. It's hard to find quality writing like yours these days. I honestly appreciate people like you! Take care!! 2022/06/05 16:12 Good web site you've got here.. It's hard to find

Good web site you've got here.. It's hard to find quality writing like
yours these days. I honestly appreciate people like you! Take care!!

# Great goods from you, man. I've understand your stuff previous to and you are just extremely wonderful. I really like what you have acquired here, really like what you are stating and the way in which you say it. You make it enjoyable and you still care 2022/06/06 14:51 Great goods from you, man. I've understand your st

Great goods from you, man. I've understand your stuff previous to and you
are just extremely wonderful. I really like what you have acquired
here, really like what you are stating and the way in which you say it.
You make it enjoyable and you still care for to keep it sensible.

I can't wait to read far more from you. This is
actually a terrific web site.

# Hello, I enjoy reading through your post. I wanted to write a little comment to support you. 2022/06/10 21:44 Hello, I enjoy reading through your post. I wanted

Hello, I enjoy reading through your post. I wanted to write a little
comment to support you.

# Wonderful goods from you, man. I have understand your stuff previous to and you're just too magnificent. I actually like what you have acquired here, really like what you are stating and the way in which you say it. You make it enjoyable and you still 2022/07/07 16:11 Wonderful goods from you, man. I have understand y

Wonderful goods from you, man. I have understand your stuff previous to and you're just too magnificent.
I actually like what you have acquired here, really like what
you are stating and the way in which you say it.
You make it enjoyable and you still care for to keep it wise.
I can not wait to read far more from you.
This is really a tremendous website.

# I visited multiple web sites except the audio feature for audio songs present at this site is actually marvelous. 2022/07/12 9:39 I visited multiple web sites except the audio feat

I visited multiple web sites except the audio feature for audio
songs present at this site is actually marvelous.

# When some one searches for his vital thing, so he/she desires to be available that in detail, thus that thing is maintained over here. 2022/07/31 19:18 When some one searches for his vital thing, so he/

When some one searches for his vital thing, so he/she desires to be available that
in detail, thus that thing is maintained over here.

# When some one searches for his vital thing, so he/she desires to be available that in detail, thus that thing is maintained over here. 2022/07/31 19:19 When some one searches for his vital thing, so he/

When some one searches for his vital thing, so he/she desires to be available that
in detail, thus that thing is maintained over here.

# When some one searches for his vital thing, so he/she desires to be available that in detail, thus that thing is maintained over here. 2022/07/31 19:21 When some one searches for his vital thing, so he/

When some one searches for his vital thing, so he/she desires to be available that
in detail, thus that thing is maintained over here.

# When some one searches for his vital thing, so he/she desires to be available that in detail, thus that thing is maintained over here. 2022/07/31 19:22 When some one searches for his vital thing, so he/

When some one searches for his vital thing, so he/she desires to be available that
in detail, thus that thing is maintained over here.

# An impressive share! I've just forwarded this onto a coworker who had been conducting a little homework on this. And he in fact bought me dinner due to the fact that I discovered it for him... lol. So let me reword this.... Thanks for the meal!! But yea 2022/08/03 16:27 An impressive share! I've just forwarded this onto

An impressive share! I've just forwarded this onto a coworker who had been conducting a
little homework on this. And he in fact bought me dinner due to the fact that
I discovered it for him... lol. So let me reword this....
Thanks for the meal!! But yeah, thanks for spending some time
to discuss this issue here on your internet site.

# It's hard to find experienced people on this subject, but you seem like you know what you're talking about! Thanks 2022/08/09 10:48 It's hard to find experienced people on this subje

It's hard to find experienced people on this subject, but
you seem like you know what you're talking about!
Thanks

# top essay writing service e91sic 2022/09/04 5:44 Charlosmox


Very good forum posts, Appreciate it. https://definitionessays.com/ how to write a literary analysis essay

# essay help chat q10nho 2022/09/08 17:07 Charlosmox


Seriously plenty of very good info! https://definitionessays.com/ how to write a scholarship essay about yourself

# Very good blog post. I absolutely appreciate this website. Stick with it! 2022/11/26 23:39 Very good blog post. I absolutely appreciate this

Very good blog post. I absolutely appreciate this website.
Stick with it!

# buy essay now p36vjg 2023/02/26 16:07 CharlesSnoff


Thanks! An abundance of data!
buy online essays https://quality-essays.com/ buying essays online safe

# phd.proposal p775kw 2023/02/27 6:54 Robertsaids


Incredible quite a lot of awesome facts.
phd proposals https://dissertationwritingtops.com/ dissertation help near me

# professional college paper writing service x37hed 2023/03/01 6:56 StevenGrelo


Amazing tons of fantastic knowledge!
best service for paper writing https://service-essay.com/ best reviewed paper writing service

# custom thesis b601tz 2023/03/03 4:58 Josephbried

You actually suggested that really well.
thesis on service quality in higher education https://writingthesistops.com/ phd thesis in service quality management

# dissertation critique t44kcc 2023/03/06 18:02 Gregorysaipt

You actually revealed this adequately.
writing an academic essay https://writeadissertation.com dissertat https://writeadissertation.com

# write essay for scholarship o48rnx 2023/03/07 1:08 EugeneSib

You revealed it superbly.
professional ghostwriting services https://customthesiswritingservice.com dissertation work https://writingthesistops.com

# professional essay writer w26vzt 2023/03/07 16:01 EugeneSib

You actually expressed it wonderfully.
editing essay services https://buycheapessaysonline.com writing a five paragraph essay https://topswritingservices.com

# essays to write r86lfz 2023/03/09 0:17 EugeneSib


With thanks. Great information.
thesis graduate https://payforanessaysonline.com how to write a good history essay https://homeworkcourseworkhelps.com

# hunter college essay v94ukj 2023/03/09 15:45 EugeneSib


Fantastic data, Thanks.
introduction to college essay https://custompaperwritersservices.com hugh gallagher college essay https://essaypromaster.com

# people write research essays in order to g856cd 2023/03/10 6:18 EugeneSib


Nicely put, Thanks a lot!
essay writing services toronto https://researchpaperwriterservices.com writing an essay for college https://writinganessaycollegeservice.com

# essay writer reddit y53srg 2023/03/10 10:08 Gregorysaipt


Nicely put, Thanks a lot!
essay pay write https://domycollegehomeworkforme.com writing services online https://englishessayhelp.com

# which is the best essay writing service i76nwx 2023/03/12 2:58 EugeneSib

You reported it wonderfully.
write my personal statement https://essaywriting4you.com funding for dissertation research https://researchpaperwriterservices.com

# the help essay questions v77gfg 2023/03/12 5:32 Gregorysaipt


Factor well utilized.!
research writing service https://essaywritingservicebbc.com help writing an essay https://service-essay.com

# my college essay n23gfp 2023/03/13 8:07 EugeneSib

You actually revealed that well!
university essay writing https://writingthesistops.com graphic organizers for writing essays https://bestpaperwritingservice.com

# how to write a community service essay c83yjj 2023/04/03 7:15 EugeneSib


Fantastic facts. Cheers.
how to write a similarities and differences essay https://dissertationwritingtops.com essaywriting service https://buyanessayscheaponline.com

# The plugins developed for WordPress 2023/05/09 23:50 Justas

The plugins developed for WordPress serve to enhance the features and functions of a WordPress website, allowing you to build your awesome and functional site https://t.me/wpigaming/648 Customise WordPress with powerful, professional and intuitive fields.

# Clava is the best 2025/05/09 13:10 Danielthash

Great news for all us

タイトル
名前
URL
コメント