The letter comes as researchers express concern that the influential site, formerly named Twitter, is a hotbed of political misinformation, while Musk -- who has endorsed Donald Trump -- appears to be swaying voters ahead of the November election by spreading falsehoods on his personal account, which has nearly 193 million followers.
Hours after President Joe Biden stepped down from the presidential race last month and endorsed Vice President Kamala Harris as the Democratic nominee, the chatbot called Grok churned out false information about ballot deadlines, which was amplified by other platforms.
"We are calling on you to immediately implement changes to... Grok to ensure voters have accurate information in this critical election year," the letter said.
The letter was signed by the secretaries of state of Minnesota, Pennsylvania, Michigan, Washington, and New Mexico. In some American states these officials are responsible for overseeing elections.
The chatbot wrongly told users that the ballot deadline had passed for nine states. The message effectively implied that Harris was not eligible to replace Biden on the ballot.
"This is false. In all nine states the opposite is true," the letter said.
"The ballots are not closed, and upcoming ballot deadlines would allow for changes to candidates listed on the ballot for the offices of president and vice president of the United States."
X did not respond to AFP's request for comment.
The letter added that Grok continued to repeat this false information –- which was amplified by multiple posts, reaching millions of people -- for more than a week until it was corrected on July 31.
"As tens of millions of voters in the US seek basic information about voting in this major election year, X has the responsibility to ensure all voters using your platform have access to guidance that reflects true and accurate information about their constitutional right to vote," the letter said.
In what is widely billed as America's first AI election in November, researchers warn that AI-enabled misinformation could be used to manipulate voters, stoking tensions in an already hyperpolarized environment.
Last week, Musk faced a firehose of criticism for sharing with his followers an AI deepfake video featuring Harris.
In it, a voiceover mimicking Harris calls Biden senile before declaring that she does not "know the first thing about running the country."
The video, viewed by millions, carried no indication that it was parody -- save for a laughing emoji. Only later did Musk clarify that the video was meant as satire.
Researchers voiced concern that viewers could have falsely concluded that Harris was deriding herself and sullying Biden.
X, which researchers say has scaled back content moderation efforts and reinstated once-banned accounts of known misinformation spreaders, has also faced criticism for stoking tensions during recent far-right riots across England.
On Sunday, Musk triggered fresh criticism for posting that "civil war is inevitable" in response to another user blaming the riots on "the effects of mass migration and open borders".
