Text Analysis API

TextRank

A sentence with its computed significance score and original ordering.

TextSignificance

Tracks word/n-gram frequencies and last-seen positions for computing intentionality scores during text analysis.

score_sentence(text::String, vocab::Dict{String,Int}) -> Float64

Compute an intentionality score for a sentence given a vocabulary of word frequencies. Higher scores indicate more distinctive/meaningful sentences.

Uses a simplified static intentionality model: words that appear a moderate number of times (not too rare, not too common) in the corpus score highest.

extract_significant_sentences(text::String; target_percent::Float64=50.0) -> Vector{String}

Extract the most significant sentences from a text, combining both running and static intentionality analysis. Returns approximately target_percent of the original sentences (merged from both methods).

text_to_n4l(text::String; chapter::String="", target_percent::Float64=50.0) -> String

Convert plain text to N4L format output. Extracts the most significant sentences and formats them as N4L notation with chapter structure, sequence markers, and extract relationships.

Returns the N4L formatted string.

reset_ngram_state!()

Initialize (or reset) the module-level n-gram frequency, location, and last-seen tracking maps. Each is a Vector of NGRAMMAX Dicts.

new_ngram_map() -> Vector{Dict{String,Float64}}

Create a fresh n-gram frequency map (vector of NGRAMMAX empty dicts).

split_into_para_sentences(text::AbstractString) -> Vector{Vector{Vector{String}}}

Split text into paragraphs → sentences → fragments. Returns a 3-level nested structure: paragraphs of sentences of fragment strings.

split_punctuation_text(s::AbstractString) -> Vector{String}

Split text on intentional separators (quotes, dashes, colons, etc.), respecting balanced parentheses.

un_paren(s::AbstractString) -> Tuple{String, Bool}

If s is wrapped in matching brackets/parens, return the inner content and true. Otherwise return trimmed s and false.

count_parens(s::AbstractString) -> Vector{String}

Split text respecting balanced parentheses/brackets/braces. Returns fragments where parenthesized groups are kept intact.

clean_ngram(s::AbstractString) -> String

Remove punctuation and lowercase an n-gram string.

excluded_by_bindings(firstword::AbstractString, lastword::AbstractString) -> Bool

Check if an n-gram starts or ends with binding words that promise to connect to adjacent content, making the n-gram a poor standalone fragment.

fractionate(frag::AbstractString, L::Int, frequency::Vector{Dict{String,Float64}}, min_n::Int) -> Vector{Vector{String}}

Extract n-grams from a text fragment using a round-robin buffer. Returns changeset where changeset[n] contains n-grams of length n.

next_word(word::AbstractString, rrbuffer::Vector{Vector{String}}) -> (Vector{Vector{String}}, Vector{Vector{String}})

Process one word through the n-gram round-robin buffer. Returns (updated rrbuffer, changeset) where changeset[n] contains the new n-grams formed at each length n.

fractionate_text(text::AbstractString) -> (Vector{Vector{Vector{String}}}, Int)

Clean, split, and build n-gram frequency/location maps from text. Updates the module-level STMNGRAMFREQ and STMNGRAMLOCA state. Returns (paragraphs, sentence_count).

fractionate_text_file(filename::AbstractString) -> (Vector{Vector{Vector{String}}}, Int)

Read file, clean, split, and build n-gram frequency/location maps. Returns (paragraphs, sentence_count).

ngram_static_intentionality(L::Int, s::AbstractString, freq::Float64) -> Float64

Compute the static significance of an n-gram string s within a document of L sentences. Intentionality = work / probability, using exponential deprecation based on SST cognitive scales (Dunbar numbers).

assess_static_intent(frag::AbstractString, L::Int, frequency::Vector{Dict{String,Float64}}, min_n::Int) -> Float64

Score a fragment by static intentionality using the n-gram round-robin buffer.

running_ngram_intentionality(t::Int, frag::AbstractString) -> Float64

Score a fragment with exponential decay based on the last-seen time of each n-gram.

intentional_ngram(n::Int, ngram::AbstractString, L::Int, coherence_length::Int) -> Bool

Determine if an n-gram is intentional (anomalous) vs ambient (repeated regular pattern). Unigrams are never intentional. Short documents are all intentional. For longer documents, checks if the distribution of inter-occurrence spacings is broad.

interval_radius(n::Int, ngram::AbstractString) -> (Int, Int, Int)

Find the minimax distances between occurrences of an n-gram (in sentences). Returns (occurrences, mindelta, maxdelta).

assess_static_text_anomalies(L::Int, frequencies, locations)

Split text n-grams into intentional (anomalous) vs ambient (contextual) parts. Returns (intent, context) — both Vector{Vector{TextRank}} of size NGRAMMAX.

assess_text_coherent_coactivation(L::Int, ngram_loc)

Global coherence analysis — separate n-grams into those that overlap across coherence intervals (ambient) and those unique to a single interval (condensate). Returns (overlap, condensate, partitions).

assess_text_fast_slow(L::Int, ngram_loc)

Running fast/slow separation by coherence intervals. For each pair of adjacent intervals, n-grams shared between them are "slow" (context), and those unique to one are "fast" (intentional). Returns (slow, fast, partitions).

coherence_set(ngram_loc, L::Int, coherence_length::Int)

Partition n-grams into coherence sets based on their occurrence locations. Returns (C, partitions) where C[n][p] is a Dict{String,Int} for n-gram size n and partition p.

extract_intentional_tokens(L::Int, selected::Vector{TextRank}, nmin::Int, nmax::Int)

Extract fast/slow parts per partition and whole-document summaries. Returns (fastparts, slowparts, fastwhole, slowwhole).

STMHistory

Tracks frequency, timing, and context key for a short-term memory fragment.

reset_stm!()

Reset the STM intentional and ambient fragment tracking state.

context_intent_analysis(spectrum::Dict{String,Int}, clusters::Vector{String})

Separate intentional (low frequency < 3) from ambient fragments. Returns (intentional::Vector{String}, ambient::Vector{String}).

update_stm_context(store::AbstractSSTStore, ambient::String, key::String, now::Int64, params)::String

Update STM context from search parameters. Extracts tokens from the params and delegates to add_context.

add_context(store::AbstractSSTStore, ambient::String, key::String, now::Int64, tokens::Vector{String})::String

Add tokens to STM tracking, prune forgotten entries, and return the combined context string of all active STM fragments.

commit_context_token!(token::AbstractString, now::Int64, key::AbstractString)

Track a token in STM. If previously seen, moves it from intentional to ambient.

intersect_context_parts(context_clusters::Vector{String})

Compute pairwise overlap between context clusters. Returns (count, uniqueclusters, overlapmatrix).

diff_clusters(l1::AbstractString, l2::AbstractString)

Return (shared, different) parts of two comma-separated context strings.

overlap_matrix(m1::Dict{String,Int}, m2::Dict{String,Int})

Return (sharedstring, differentstring) representing overlap and unique parts of two token frequency maps.

get_context_token_frequencies(fraglist::Vector{String}) -> Dict{String,Int}

Build a frequency map from a list of comma-separated context strings.

get_node_context(store::AbstractSSTStore, node::Node) -> Vector{String}

Get the context strings attached to a node via the empty arrow ghost link. Returns a vector of context labels parsed from the comma-separated context string.

get_node_context_string(store::AbstractSSTStore, node::Node) -> String

Get the context string from a node's ghost link (empty arrow, LEADSTO type). The context is stored as an incoming link with the "empty" arrow.

context_interferometry(context_clusters::Vector{String}) -> Nothing

Deprecated/placeholder. In the original Go source this function was deleted (body replaced with // deleted). Retained here as a no-op stub for API compatibility. Use intersect_context_parts and diff_clusters instead.

do_nowt(then::DateTime) -> Tuple{String, String}

Convert a DateTime to a semantic time representation. Returns (whendescription, timekey) where:

• when_description is a human-readable semantic time string
• time_key is a compact key for database/context use

get_time_context() -> Tuple{String, String, Int64}

Returns (contextstring, timekey, unix_timestamp) for the current time.

season(month::AbstractString) -> Tuple{String, String}

Returns (northernhemisphereseason, southernhemisphereseason) for the given month name.

get_time_from_semantics(speclist::Vector{String}, now::DateTime) -> DateTime

Parse a semantic time specification (Day3, Hr14, Mon, etc.) into a DateTime. The first element of speclist is ignored (it's typically a command prefix).

LogFormat

Enum for supported log formats.

LogParseConfig

Configuration for log parsing into SST.

default_log_config() -> LogParseConfig

Return default configuration for log parsing (syslog format).

parse_syslog_line(line::String) -> NamedTuple

Parse a syslog-format line: "Jan 15 10:30:45 hostname service[pid]: message"

parse_json_log_line(line::String; config::LogParseConfig=default_log_config()) -> NamedTuple

Parse a JSON log line.

parse_csv_log(text::String; delimiter::Char=',', header::Bool=true) -> Vector{NamedTuple}

Parse CSV/TSV log data. First row is treated as header when header=true.

parse_log_to_sst!(store::MemoryStore, text::String;
                  config::LogParseConfig=default_log_config()) -> Dict{String,Int}

Parse log text and create SST nodes for each log entry.

• Each entry becomes a node in the configured chapter
• Sequential entries are linked via LEADSTO (then) if link_sequential=true
• Log level, source, and extracted patterns become EXPRESS (note) annotations
• Timestamp becomes a timeline annotation

Returns statistics: entries_parsed, nodes_created, links_created.

EntitySuggestion

A suggested entity extracted from text.

LinkSuggestion

A suggested relationship between two entities.

TextBreakdown

Complete analysis of a text passage.

identify_entities(text::String) -> Vector{EntitySuggestion}

Extract named entities and concepts from text using heuristic NLP patterns.

suggest_links(text::String, entities::Vector{EntitySuggestion}) -> Vector{LinkSuggestion}

Given extracted entities, suggest SST relationships based on textual proximity and verb/preposition patterns.

propose_structure(text::String; chapter::String="default") -> TextBreakdown

Full text analysis: extract entities, suggest links, generate N4L.

breakdown_to_n4l(tb::TextBreakdown; chapter::String="default") -> String

Convert a TextBreakdown into N4L notation.